Models of Translation Competitions

What do we want to learn from a translation competition and how do we learn it with confidence? We argue that a disproportionate focus on ranking competition participants has led to lots of different rankings, but little insight about which rankings we should trust. In response, we provide the first framework that allows an empirical comparison of different analyses of competition results. We then use this framework to compare several analytical models on data from the Workshop on Machine Translation (WMT). 1 The WMT Translation Competition Every year, the Workshop on Machine Translation (WMT) conducts a competition between machine translation systems. The WMT organizers invite research groups to submit translation systems in eight different tracks: Czech to/from English, French to/from English, German to/from English, and Spanish to/from English. For each track, the organizers also assemble a panel of judges, typically machine translation specialists.1 The role of a judge is to repeatedly rank five different translations of the same source text. Ties are permitted. In Table 1, we show an example2 where a judge (we’ll call him “jdoe”) has ranked five translations of the French sentence “Il ne va pas.” Each such elicitation encodes ten pairwise comparisons, as shown in Table 2. For each competition track, WMT typically elicits between 5000 and 20000 comparisons. Once the elicitation process is complete, WMT faces a large database of comparisons and a question that must be answered: whose system is the best? Although in recent competitions, some of the judging has also been crowdsourced (Callison-Burch et al., 2010). The example does not use actual system output. rank system translation 1 bbn “He does not go.” 2 (tie) uedin “He goes not.” 2 (tie) jhu “He did not go.” 4 cmu “He go not.” 5 kit “He not go.” Table 1: WMT elicits preferences by asking judges to simultaneously rank five translations, with ties permitted. In this (fictional) example, the source sentence is the French “Il ne va pas.” source text sys1 sys2 judge preference “Il ne va pas.” bbn cmu jdoe 1 “Il ne va pas.” bbn jhu jdoe 1 “Il ne va pas.” bbn kit jdoe 1 “Il ne va pas.” bbn uedin jdoe 1 “Il ne va pas.” cmu jhu jdoe 2 “Il ne va pas.” cmu kit jdoe 1 “Il ne va pas.” cmu uedin jdoe 2 “Il ne va pas.” jhu kit jdoe 1 “Il ne va pas.” jhu uedin jdoe 0 “Il ne va pas.” kit uedin jdoe 2 Table 2: Pairwise comparisons encoded by Table 1. A preference of 0 means neither translation was preferred. Otherwise the preference specifies the preferred system.